Fraudulent activity detection at a barcode scanner by verifying visual signatures

ABSTRACT

System and method for detecting a fraudulent activity at a barcode scanner is disclosed. The method issues an alert when the fraudulent activity is confirmed by comparing the visual signature of the item being transacted over the checkout terminal to the model visual signature. The model visual signature is obtained by averaging the collection of visual signature of the item gathered over a period of time. A human validation via a remote processor is employed to confirm the fraudulent activity verified by a computer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation utility patent application whichclaims the benefit to and priority from U.S. patent application Ser. No.14/591,883 filed on Jan. 7, 2015.

BACKGROUND Field of the Invention

The present invention relates generally to a security system and methodto prevent fraudulent activities occurring at a checkout terminal. Moreparticularly, it relates to a method and system for detecting afraudulent activities using a barcode scanner and a camera. The presentinvention, in part, provides a solution to detect a fraudulent activityknown as “ticket switching”.

Description of Related Art

Retailer lose billions of dollars annually due to the theft and improperscanning of the merchandise they are selling. This can occur in numerousways, due to both employees stealing from the retailers as well as dueto customers stealing. Of all the methods used which result in loss tothe retailer, a significant portion occurs at the checkout itself.Improper scanning, “sweethearting”, leaving items unscanned in carts andshopping baskets are just some of the ways in which customers andcashiers alike contribute to inventory “shrinkage” which results in aloss for the retailer.

One way in particular that a cashier or a customer can engage infraudulent activity is through a process called ticket switching. Ticketswitching is the process of replacing the barcode of one item with abarcode of another item of lesser value. (This is enabled by means assimple as pre-printing the less expensive barcode on a sticker beforeentering the store.) For instance, one could replace the barcode on thebox of an expensive cordless drill with the barcode of an inexpensivetool. Thus, when the item is rung up for sale, the less-priced item isrecorded instead. If it is the customer engaging in the fraud, thecustomer may hope that the cashier does not notice the difference.

Likewise, a customer could use a self-checkout register and bypass thatmode of uncertainty altogether. If it is the cashier engaged in thefraud, often in collusion with the customer, he or she may have variousother ways of engaging in the act of ticket switching. They may havebarcodes ready on the side which are scanned in place of the certainexpensive items. For instance, in a grocery store, the cashier may wishto ring up an expensive meat item for a friend by scanning in aninexpensive item, like a can of beans, in place of the meat barcode.Likewise, a cashier may stick the barcode of an inexpensive item to hisor her wrist such that the wrist barcode gets scanned while the cashiermoves a more expensive item (with its barcode un-viewable by thescanner) across the scanner.

Such fraudulent activities significantly contribute to the inventoryshrinkage and subsequent loss of revenue for retailers. Therefore, whatis needed is a system and method that protects retailers' assets frominadvertent or deliberate loss by preventing a fraudulent activity at abarcode scanner.

SUMMARY

The subject matter of this application may involve, in some cases,interrelated products, alternative solutions to a particular problem,and/or a plurality of different uses of a single system or article.

In one aspect, a method for detecting a fraudulent activity at acheckout terminal is provided. The checkout terminal may comprise acomputer. The method may begin with the computer detecting an itemidentifier number of an item from the barcode with a barcode scanner,where the barcode is affixed to the item. The barcode scanner may be incommunication with the computer. Further, the camera may capture animage of the item and be positioned to take an image of the item. Thecamera may be in communication with the computer. The image capture bythe camera may comprise the barcode and a surrounding of the barcode.

The method may carry on by the computer obtaining a visual signaturefrom the image. Finally, the computer may determine a similarity betweenthe visual signature and a model visual signature to verify thefraudulent activity, by comparing the visual signature obtained from theimage to the model visual signature linked to the item identifiernumber. The model visual signature may represent an expected visualsignature of the item, where the model visual signature is stored at astorage unit being accessible to the computer.

In another aspect, a checkout terminal for detecting a fraudulentactivity is provided. The checkout terminal may comprise a computer, abarcode scanner, a camera, and a storage unit in communication with thecheckout terminal via a network. The barcode scanner may be incommunication with the computer and positioned to detect an itemidentifier number of an item from a barcode that is affixed to the item.The camera may be in communication with the computer and positioned tocapture an image of the item, where the image comprises the barcode anda surrounding of the barcode.

The checkout terminal may be configured to obtain a visual signaturefrom the image. Further, the checkout terminal may be configured todetermine a similarity between the visual signature and a model visualsignature to verify the fraudulent activity, by comparing the visualsignature obtained from the image to the model visual signature linkedto the item identifier number. The model visual signature may representan expected visual signature of the item, where the model visualsignature may be stored at the storage unit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides an embodiment of the checkout terminal.

FIG. 2 provides a flowchart showing an embodiment of the method fordetecting a fraudulent activity at the checkout terminal.

FIG. 3 provides a flowchart of an exemplary embodiment overviewing thesteps involved in obtaining the model visual signature.

FIG. 4 provides an exemplary embodiment of the barcode scanner.

FIG. 5 provides examples of image distortions.

FIG. 6 provides an exemplary embodiments of correcting the scaledistortion.

FIG. 7 provides an exemplary embodiments of correcting the rotationaldistortion.

FIG. 8 provides an exemplary embodiments of correcting the affinedistortion.

FIG. 9 provides an exemplary embodiments of correcting projectivedistortion.

FIG. 10 provides an exemplary embodiment of the visual signature.

DETAILED DESCRIPTION

The detailed description set forth below in connection with the appendeddrawings is intended as a description of presently preferred embodimentsof the invention and does not represent the only forms in which thepresent invention may be constructed and/or utilized. The descriptionsets forth the functions and the sequence of steps for constructing andoperating the invention in connection with the illustrated embodiments.

Generally, the present invention concerns a system and method fordetecting a fraudulent activity at checkout terminal having a barcodescanner. Specifically, the present invention provides a solution todetect and prevent a fraudulent activity commonly known as “ticketswitching”. Ticket switching is a process of replacing a barcode of anitem with another barcode of another item of lesser value. Often times,the barcode of an item is altered by attaching another barcode ofanother item to cover up the existing barcode on the item. Thus ticketswitching or barcode replacing still leaves other elements or featuresof the item intact to that item. For example, packaging of the itemcarries various visual features such as color, shape, and characters, inaddition to the barcode.

As such, the present invention provides a system and method that detectsthe fraudulent activity (for example, ticket switching) by identifying apartial or substantial change of the item's visual features. The systemand method provided herein may be employed to detect other similar typesof fraudulent activity that may occur at the checkout terminal. Thechange can be verified by comparing the item's visual features linked tothe barcode with the item's visual features obtained during atransaction of such item at the checkout terminal. Once the barcode isswitched to another item, the corresponding visual features are likelyto be altered, thus the present invention provides a system and methodthat is capable of detecting such changes by verifying the visualfeatures of the item in question. Each of the items being scanned overthe checkout terminal may be severally verified by the system and methodprovided herein.

Further, the present invention provides a system and method thatprovides alerts once the fraudulent activity is detected at the checkoutterminal. The alerts may be given in many different ways which aredisclosed in the following descriptions.

Camera contemplated herein may include, but are not limited to, DSLR,non-SLR digital cameras (e.g., but not limited to, compact digicams andSLR-like bridge digital cameras (also known as advanced digitalcameras), and SLR-like interchangeable lens digital cameras), as well asvideo recorders (e.g., but not limited to, camcorders, analog camerasand IP cameras, and the like; a device that can provide a video feed ofany duration, such as a DVR; a portable computing device having acamera, such as a tablet computer, laptop computer); and the like.

Storage unit contemplated herein may store data in the format including,but are not limiting to, XML, JSON, CSV, binary, over any connectiontype: serial, Ethernet, etc. over any protocol: UDP, TCP, and the like.

Image contemplated herein may be any digital image format capable ofbeing interpreted by a computer or computing device. Examples of imagefiles contemplated herein include, but are not limited to JPEG, GIF,TIFF, PNG, Bitmap, RAW, PNM, WEBP, and the like.

Computer contemplated herein may include, but are not limited to,desktop computers, laptop computers, tablet computers, handheldcomputers, smart phones and other cellular phones, and similar internetenabled mobile devices, digital cameras, a customized computing deviceconfigured to specifically carry out the methods contemplated in thisdisclosure, and the like.

Barcode contemplated herein may be of any type of machine-readablemedium, including but not limited to a barcode, a QR code,two-dimensional bar code, a prescribed font, optical characterrecognition (OCR) characters, Radio Frequency Identification (RFID),Near-Field Communication (NFC), Bluetooth technology, alphanumericcharacters, non-alphanumeric characters, symbols, facial recognition andthe like.

Network contemplated herein may include, for example, one or more of theInternet, Wide Area Networks (WANs), Local Area Networks (LANs), analogor digital wired and wireless telephone networks (e.g., a PSTN,Integrated Services Digital Network (ISDN), a cellular network, andDigital Subscriber Line (xDSL)), radio, television, cable, satellite,and/or any other delivery or tunneling mechanism for carrying data.Network may include multiple networks or sub-networks, each of which mayinclude, for example, a wired or wireless data pathway. The network mayinclude a circuit-switched voice network, a packet-switched datanetwork, or any other network able to carry electronic communications.Examples include, but are not limited to, Picture Transfer Protocol(PTP) over Internet Protocol (IP), IP over Bluetooth, IP over WiFi, andPTP over IP networks (PTP/IP).

Barcode scanner contemplated herein may include, but are not limited to,a laser-based scanner, an optical-based scanner, and the like.

Video contemplated herein may be any data format capable of beinginterpreted by a computer or computing device. Examples of image filescontemplated herein include, but are not limited to MP4, AVI, MPG, M4V,FLV, MOV, WMV, VOB, and the like.

The system for detecting a fraudulent activity at a checkout terminal isprovided. The system may comprise a camera, a barcode scanner, acomputer with one or more processors, and a storage unit accessible bythe computer via a network. The system may comprise one or morecomputers or computerized elements in communication working together tocarry out the different functions of the system. The inventioncontemplated herein further may comprise non-transitory computerreadable media configured to instruct a computer or computers to carryout the steps and functions of the system and method, as describedherein.

In one embodiment, the system may be implemented with the checkoutterminal. The checkout terminal for detecting a fraudulent activity maycomprise a computer, a barcode scanner, a camera, and a storage unit.

The barcode scanner may detect an item identifier number of an itembeing transacted over the barcode scanner by reading the barcode affixedto the item. The barcode scanner may be operable to communicate with thecomputer for further processing of the item identifier number. Thebarcode scanner may be of any type capable of sensing or reading thebarcode in order to scan and record items being transacted over thecheckout terminal.

In one embodiment, the barcode scanner may be a laser-based scanner. Inanother embodiment, the barcode scanner may be an optical-based scanner.The optical-based scanner utilizes an image of the item to identify thebarcode.

The item identifier number may be, for example, a stock keeping unitnumber (SKU) or product lookup number (PLU).

The camera may be positioned to capture an image of the item beingtransacted over the barcode scanner. The image may be captured such thatit includes the barcode and a surrounding of the barcode. The camera maybe operable to communicate with the computer for further processing ofthe captured image. The camera may be further capable of recording avideo or a snippet of video while the checkout terminal is operationalor while the checkout terminal is scanning multiple items during thetransaction of the items. The snippet of video may comprise at least oneimage of the item. For example, the snippet of video may comprisemultiple images or frames showing the item viewed from various angles.

In one embodiment, the camera and the barcode scanner may be integratedinto the optical-based scanner. The optical-based scanner uses a digitalimaging sensor or sensors to read barcodes in order to scan and recordthe item being transacted at the checkout terminal. As such, theoptical-based scanner comprises the camera integrated therein.

The checkout terminal for detecting a fraudulent activity may comprisemultiple cameras situated to capture the image of the item at multipleangles. For instance, the item may be affixed with multiple barcodes, insuch instance, multiple cameras may work together to capture multipleimages including the multiple barcodes affixed to the item.

In another embodiment, the camera may be positioned nearby the barcodescanner, such that detection of the barcode by the barcode scanner andimage capture by the camera occur substantially at the same time.

In yet another embodiment, the camera may be an overhead camera whichare positioned to provide a better view of the item during thetransaction of the item at the checkout terminal.

The camera may be utilized to serve different functions, not limiting tothe operational and functional embodiments disclosed herein. Similarly,the camera may be employed to serve more than one function, for example,to prevent other types of fraudulent activities that happen at thecheckout terminal.

The storage unit may be operable to communicate with the checkoutterminal and its components. The storage unit may receive images beingtaken by the camera for further verification of the fraudulent activity.The storage unit also may receive item identifier numbers from thebarcode scanner for further verification of the fraudulent activity.

In particular, the storage unit may be accessible by the computer forcomparing the image of the item captured during an instance of thetransaction to an image that is expected by the item identifier numberof the item of that instance. The steps and functions of the method fordetecting a fraudulent activity utilizing the system and/or checkoutterminal described above are further discussed herein.

Turning now to FIG. 1, an embodiment of the checkout terminal isprovided. The checkout terminal 101 comprises the barcode scanner 103integrated with the camera 104. Further, the checkout terminal 101comprises the computer 102 having one or more processor, and a display105. The barcode scanner 103 and the camera 104 are in communicationwith the computer 102. The checkout terminal 101 is in communicationwith the storage unit 107 via a network 106. The display may provide acomputerized user interface for an attendant to monitor the transactionat the checkout terminal.

In one embodiments, the display may allow input and output relating tothe steps and actions of the system and method described herein.

In another embodiment, the display may provide access to the datastorage unit 107 via the network 106.

In some embodiments, a remote processor 109 may be further incommunication with the checkout terminal via the network. The remoteprocessor 109 may further verify the fraudulent activity from a remotelocation or by employing a different method of verification. Theverification of the fraudulent activity identified by the computer maybe necessary to be confirmed, so that a false-positive verification ofthe fraudulent activity can be prevented. The remote processor 109, forexample, may receive the image or a video from the camera for a humanvalidation of the fraudulent activity.

In some embodiments, the checkout terminal 101 may be in furthercommunication with an alert system 108. The alert system 108 may issuean alert when the fraudulent activity is confirmed by the computer. Inone embodiment, the alert may be a visual alarm, such as light or othervisual signals. In another embodiment, the alert may be an audiblealarm. In yet another embodiment, the alert may be an alert messagepresented on the display. The display may be monitored by a cashier atthe checkout terminal or an attendant in case the checkout terminal is aself-checkout terminal. Examples of the alert message may include, butare not limited to, a text message, email, or other electronic messages.

In a further embodiment, the alert may be initiated by halting theoperation of the checkout terminal or the self-checkout terminal. Uponhalting of the operation, a review of the alert may be required by theattendant. The review may be conducted by the attendant physicallyverifying the alert. The review also may be conducted by the attendantreceiving a snippet of video sent to the checkout terminal or anattendant's remote processor. The snippet of video may be provided bythe camera which includes at least a portion of the video recording theitem during the transaction relevant in time to the fraudulent activityin question, possibly up to and including the entire transaction up tothat point, in order for the attendant to quickly review what hasoccurred to trigger the alert.

In a further embodiment, the snippet of video may comprise at least oneimage of the item. For example, the snippet of video may comprisemultiple images or frames showing the item viewed from various angles,making the review by the attendant easier. The snippet of video also maycomprise the image captured by the camera. Further, the attendant mayreview the transaction, correct any errors, and allow the transaction tocontinue once the alert is resolved.

A method for detecting a fraudulent activity at a checkout terminal isprovided. The method may be employed by the system and/or the checkoutterminal provided above.

The method for detecting a fraudulent activity may begin with detectingthe item identifier number of the item being transacted over thecheckout terminal with the barcode scanner. The item identifier numberof the item may be detected by reading the barcode affixed to the item.The barcode scanner may be operable to communicate with the computer.The image of the item may be captured with the camera positioned at thecheckout terminal. In one embodiment, the image may be captured suchthat it includes the barcode of the item and the surrounding of thebarcode.

Further, the computer may obtain a visual signature from the image. Inone embodiment, the interesting point may be automatically chosen by theinterest operator. The interest operator is an image processingfunctions that detect interesting locations within an image.

In another embodiment, the visual signature may be obtained by detectingan interesting point from the image using an interest operator. Theinteresting point may be located in the surrounding of the barcodecaptured in the image. The visual signature may be obtained by detectinga plurality of interesting points. The interest operator may be anycombination of a corner detector, an edge detector, and an area of hightexture. These interest operator provides visual features unique to theitem in addition to the barcode. In yet another embodiment, the visualfeatures may include scale-invariant feature transform (SIFT) or speededup robust features (SURF) features, which are rotationally and scaleinvariant. Other visual features, may include, but are not limited to,affine invariant features, histogram based approaches, image-basedfeatures (e.g. image patches), and the like.

The computer may determine a similarity between the visual signature anda model visual signature to verify whether the item being analyzed hasbeen fraudulently tempered. The similarity may be determined bycomparing the visual signature from the image to the model visualsignature linked to the item or the item identifier number. The modelvisual signature represents an expected visual signature of the itemwhich may be accessible from the storage unit.

In one embodiment, the model visual signature may be obtained bycollecting a group of visual signatures from the item. As the item istransacted multiple times over the checkout terminal in a period oftime, the visual signatures from each of the transactions of the itemmay be collected. Once collected, a probabilistic model of the group ofvisual signatures may be calculated to obtain the model visual signaturethat can be compared against the visual signature of the item as theitem is transacted post obtainment of the model visual signature.

In one embodiment, the probabilistic model may be a statistical modelretrieved from the group of visual signatures. In general, it isunderstood that “model” can refer to any piece of data, statistical orotherwise, that is used to compare the current instance of an item'svisual signature with what is expected of that item (model visualsignature). As such, in some embodiments, the probabilistic model may bea nonparametric probabilistic model.

The nonparametric probabilistic model of the group of visual signaturemay be calculated to obtain the model visual signature that can becompared against the visual signature of the item as the item istransacted post obtainment of the model visual signature. By way of anon-limiting example, a nonparametric probabilistic model can beconstructed by collecting ten of the visual signatures from the mostrecent transactions of the item. The similarity of a new visualsignature from a subsequent item can be determined by first comparingthe new visual signature to each of the ten visual signatures within thenonparametric probabilistic model. As a result, ten similarities can begathered from each of the ten comparisons. The least differentsimilarity among the ten similarities can then be used as the similarityfor the subsequent item. Thus, the nonparametric probabilistic model isthe visual signature corresponding to the least different similarityamong the ten. Various other similarity measures can be applied as well,including median, mean, etc. Furthermore, various approaches tocollecting the models can also be employed in a nonparametric strategy.

In another embodiment, statistically, a probability that the visualsignature is sampled from the model visual signature may be determinedby using a likelihood estimate. This is measured mathematically asP(D|M), probability of observing this data, D, given the model M. Thisconstitutes a most likely, or ML, estimate. This is related in manycases to taking a simple difference of the visual signature incomparison to the model visual signature. In the case where the modelvisual signature may be just an average of the appearances of variousvisual signatures seen over time, and the visual signatures are imagepatches, this is exactly what the likelihood estimate consists of. Thelikelihood estimate can then be compared with a threshold value. It canalso be compared with likelihood estimate taken from one or more of theother model visual signatures stored in the data storage unit in orderto determine which model visual signature the current visual signatureis most likely to have been sampled from. This is related to the maximuma-posteriori estimate of the data,

${P\left( M \middle| D \right)} = {\frac{{P\left( D \middle| M \right)}{P(M)}}{\sum_{i}{{P\left( D \middle| M_{i} \right)}{P\left( M_{i} \right)}}}.}$From this, one can see the ML estimate, P(D|M), is used in thiscalculation, but it is used in conjunction with calculation of thelikelihood estimate taken from all the other visual signature modelsover which one wishes to compare it with.

In yet another embodiment, the probabilistic model may be obtained bytaking an average of the group of visual signatures which takes anaverage of the appearances of various visual signatures collected overthe period of time.

In a further embodiment, the model visual signature may be approximatedaccording to Gaussian distribution. In this embodiment, the similaritymay be examined by a threshold value, where the threshold valuerepresents a boundary of acceptable similarity range within the Gaussiandistribution gathered from the group of visual signatures. Thus, thethreshold value may set a maximum allowed difference in the similarity.

A size of the group of visual signature required to obtain an accuratemodel visual signature may vary. Similarly, the size of the group ofvisual signatures required to begin measuring the similarity may vary.In one embodiment, the size of the group of visual signatures may bedetermined by a measure of variances among the collected group of visualsignatures. The measure of variances may represent how self-similar thecollected visual signatures are as the item is transacted multiple timesover the checkout terminal in a period of time. If they are sufficientlysimilar or the measure of variance is low, then a smaller size of thegroup may be needed to build an effective model visual signature.Likewise, if the collected visual signatures are quite dissimilar or themeasure of variance is high, it may take more samples in order to buildan effective model visual signature.

In one embodiment, the measure of variance to determine the size of thegroup of visual signatures can be accomplished by using online methodsof learning the parameters of a Gaussian distribution, using themagnitude of the variances to determine the spread or cohesiveness ofthe collected group of visual signatures.

In another embodiment, the size of the group of visual signatures may bedetermined by setting a predetermined size of the group of visualsignatures. In this embodiment, the model visual signature is obtainedand applied to measure the similarity, once the predetermined size ofthe group of visual signatures are collected.

A further maintenance of the model visual signature may be continuouslyundertaken as more of the visual signature of the same item is obtainedover a period of time. In one embodiment, the visual signature obtained,at each instance of transactions of the item over the checkout terminal,may be added to the group of visual signatures to update the modelvisual signature continuously. This embodiment may only apply ifverification of the fraudulent activity yields false, in other words,the visual signature matches the model visual signature.

In another embodiment, the computer may reset the group of visualsignatures. In case where the model visual signature is deemedinaccurate the reset may be needed. In this embodiment, the computer maybeing to reset the group of visual signatures when a predeterminednumber of occasions indicating a mismatch between the visual signatureand the model visual signature has been observed. The reset may begin bydiscarding the current model visual signature and rebuilding the groupof visual signatures. The reset may begin to take place, while thecurrently active model visual signature continues to apply to determinethe similarity. Once an accurate and new model visual signature isobtained from the reset, the currently active model visual signature maybe replaced with the one created by the reset.

The method for detecting a fraudulent activity may further involve thealert system. When the visual signature of the item does not match themodel visual signature, in other words, when the similarity determinesthat the visual signature is not within the acceptable similarity range,the computer may issue an alert. The similarity may be verified againstthe threshold value. The threshold value may represent represents aboundary of acceptable similarity range within the Gaussian distributiongathered from the group of visual signatures. The threshold value alsomay be an absolute measurement of similarity which sets the maximumallowed difference against the model visual signature.

The method for detecting a fraudulent activity may further involve theremote processor. The remote processor may further verify orpreliminarily verify the fraudulent activity prior to issuing the alarm.

In one embodiment, the remote processor may receive the image for ahuman validation of the fraudulent activity. Once received, the humanvalidation may take place by a human to determine the veracity of thefraudulent activity at a remote location having access to the remoteprocessor. The result of the human validation may be sent back to thecomputer.

In another embodiment, the snippet of video may be sent to the remoteprocessor by the computer for the human validation. The snippet of videomay comprise at least one image of the item. For example, the snippet ofvideo may comprise multiple images or frames showing the item viewedfrom various angles.

In yet another embodiment, the human validation may be requested to theremote processor by the computer based on a confidence score. Theconfidence score may represent a degree of difference between the visualsignature and the model visual signature.

The human validation can be considered an extension of the matchingprocess itself rather than a distinct step. Thus, if the optional humanvalidation step is undertaken, then the result of that decision is takenas the ultimate result of the verification process. This is particularlyuseful in the case in which the verification process carried by thecomputer as described above shows a negative match, but the humanvalidation step shows a positive match. In this case, not only is afalse alert prevented, but the visual signature of such instance canthen be used to update the model visual signature.

The method for detecting a fraudulent activity may further comprisecorrecting the image into a canonical view. In order to facilitate thecreation of the visual signature, correcting image distortions in theimage may be necessary. Correcting the image distortions may help toextract and compare the visual features or the interesting points withinthe surrounding of the barcode, by creating a uniform perspective withwhich to base the comparison against the model visual signature.Examples of image distortions may include, but are not limited to, scaledistortion, rotational distortion, projective distortion, and barreldistortion. The canonical view may be obtained by applying a transformto compensate for the image distortions.

In one embodiment, the transform may be identified by comparing a sizeand shape of the barcode in the image to a predetermined reference sizeand shape of the barcode. The predetermined reference size and shapecorresponding to the identifier number may be stored at the storage unitand accessible to the computer.

In another embodiment, the size and shape of the barcode in the imagemay be determined by fitting a bounding box around the barcode withinthe image. The bounding box need not be axis-aligned.

In yet another embodiment, the size and shape of the barcode in theimage may be determined by identifying at least two corner points of thebarcode and their relations between the two within the image.

In yet another embodiment, the size and shape of the barcode in theimage may be determined by calculating a parameter of a plane on whichthe barcode resides.

The barcode may be affixed to the item in more than one location. Itempackaging often contains more than one barcodes. This allows barcodescanners to scan items when the item can be in many differentorientations with respect to the barcode scanner or the camera. Thus,multiple model visual signature may be required to accommodate the manydifferent orientations. This may be achieved by using typicalmulti-modal statistical methods.

By way of example, if an item has two barcodes on its packaging, themodel visual signature obtained will be bimodal. Likewise, if it hasthree barcodes, the model visual signature obtained will be trimodal.There are many number of methods known in the art to achieve this,including mean-shift, minimum description length, expectationmaximization approaches, etc.

Furthermore, a number can simply be chosen for how “modal” thedistribution should be. For this, a simple k-means classification schemecan be used, where k is the chosen number of modes, or barcodes, on theitem. In many scenarios, this is often the better choice, as often timesthere may be multiple barcodes on the packaging, though one or two mayget the majority of the scans. Furthermore, selecting a value of k thatis only slightly too high does not typically create poor models, as theone or two superfluous modes are simply starved out. Again, theseapproaches to statistical modeling would be apparent to those havingordinary skill in the art.

FIG. 2 shows a flow chart of an embodiment of the method for detecting afraudulent activity at the checkout terminal. In step 201, the computerdetects an item identifier number of an item. Then the camera capturesan image of the item 202. From the captured image, the computer furtherobtains a visual signature from the image 203. The visual signatureobtained from step 203 is then compared with the model visual signatureto determine the similarity between the visual signature and the modelvisual signature 204. At 205, the fraudulent activity is verified bymatching the visual signature and the model visual signature. If theymatch the computer passes 206 the item being transacted. If not, thecomputer issues the alert 207.

FIG. 3 shows a flow chart of an exemplary embodiment overviewing thesteps involved in obtaining the model visual signature. At 301, thegroup of visual signatures is collected by the camera and stored in thestorage unit. Once the group of visual signatures is collected, thecomputer creates the probabilistic model 302. Then, the model visualsignature is obtained 303 by various methods described above. Theobtained model visual signature may further be updated 304 by addingadditional visual signature to the group or be reset 305 if the obtainedmodel visual signature is in accurate.

FIG. 4 illustrates an exemplary embodiment of the barcode scanner. Inthis exemplary embodiment, the barcode scanner 103 comprises two glasspanels 401 402 positioned to face the item at the bottom or the side.The glass panels 401 402 can hold the camera therein to capture theimage of the item as its barcode is scanned through either of the twoglass panels 401 402.

FIG. 5 shows examples of image distortions. The image shown in 501represents the canonical view of the image containing the barcode andthe surrounding at its predetermined reference size and shape. In thisimage the barcode reads from left to right. As shown in 502, the imagemay contain the scale distortion. An example of the rotationaldistortion is shown in 503. Lastly, example of the projective distortionis shown in 504.

FIG. 6-9 illustrate exemplary embodiments of correcting the image intothe canonical view.

In FIG. 6, the image 601 is distorted with a scale distortion. Tocorrect the scale distortion from the image, a scale transform can beapplied to the image 601. At 602, the scale transform is identified byfirst fitting the bounding box 603 around the barcode within the imageto determine the size and shape of the barcode in the image. Then thesize and shape of the barcode in the image is compared to thepredetermined reference size and shape of the barcode corresponding tothe item identifier number to calculate the scale transform. The imageat 604 shows the image with the scale distortion removed by applying thescale transform to the image 601.

In FIG. 7, the image 701 is distorted with a rotational distortion. Tocorrect the rotational distortion from the image, a rotational transformcan be applied to the image 701. At 702, the rotational transform isidentified by first fitting the bounding box 603 around the barcodewithin the image to determine the size and shape of the barcode in theimage. Then the size and shape of the barcode in the image is comparedto the predetermined reference size and shape of the barcodecorresponding to the item identifier number to calculate the rotationaltransform. The image at 703 shows the image with the rotationaldistortion removed by applying the rotational transform to the image701.

In FIG. 8, the image 801 is distorted both with a scale distortion and arotational distortion. The combination of the scale distortion and therotational distortion results in the affine distortion. To correct theaffine distortion, both rotation and scale, from the image, the affinetransform can be applied to the image 801. At 802, the affine transformis identified by first fitting the bounding box 603 around the barcodewithin the image to determine the size and shape of the barcode in theimage. Then the size and shape of the barcode in the image is comparedto the predetermined reference size and shape of the barcodecorresponding to the item identifier number to calculate the affinetransform. The image at 803 shows the image with the affine distortionremoved by applying the affine transform to the image 801.

In FIG. 9, the image 901 is distorted with a perspective distortion. Tocorrect the perspective distortion, the parameter of a plane on whichthe barcode resides on the image is calculated. This process is alsocalled ground plane rectification. Once the barcode is fitted with thebounding box 603 at 902, the projective transform can be obtained bycomparing the size and shape of the barcode in the image to thepredetermined reference size and shape of the barcode corresponding tothe item identifier number. The projective transform is applied to theimage at 901 to arrive at the canonical view 903.

FIG. 10 illustrates an exemplary embodiment of the visual signature. Thedetected barcode 1008 obtained from the image by either the camera orthe optical-based scanner is identified. In the surrounding of thebarcode 1009, a plurality of interesting points (1001 1002 1003 10041005 1006 1007) are detected by the computer. In this exemplaryembodiment, the visual signature comprises the plurality of interestingpoints, the barcode, and locations of each of the plurality ofinteresting points relative to the barcode.

While several variations of the present invention have been illustratedby way of example in preferred or particular embodiments, it is apparentthat further embodiments could be developed within the spirit and scopeof the present invention, or the inventive concept thereof. However, itis to be expressly understood that such modifications and adaptationsare within the spirit and scope of the present invention, and areinclusive, but not limited to the following appended claims as setforth.

Those skilled in the art will readily observe that numerousmodifications, applications and alterations of the device and method maybe made while retaining the teachings of the present invention.

What is claimed is:
 1. A method for detecting a fraudulent activity at acheckout terminal, the checkout terminal comprising a computer, themethod operated via the computer, comprising the steps of: detecting anitem identifier number of an item by scanning a barcode with a barcodescanner, the barcode being affixed to the item, wherein the barcodescanner is in communication with the computer; capturing an image of theitem with a camera, the camera positioned to take the image of the item,the camera being in communication with the computer, wherein the imagecomprises the barcode and a surrounding of the barcode; obtaining avisual signature from the image, wherein the visual signature isobtained based on the barcode and an interesting point located in thesurrounding of the barcode; determining a similarity between the visualsignature and a model visual signature by comparing the visual signatureobtained from the image to the model visual signature associated withthe item identifier number, the model visual signature representing anexpected visual signature of the item, wherein a plurality of modelvisual signatures is stored at a storage unit being accessible to thecomputer, each of which being associated with an item identifier number;and detecting the fraudulent activity based on the determinedsimilarity.
 2. The method of claim 1 wherein the barcode scanner is atleast one of an optical-based scanner and a laser-based scanner, theoptical-based scanner comprising the camera.
 3. The method of claim 1wherein the step of obtaining a visual signature from the imagecomprises: detecting the interesting point from the image using aninterest operator, wherein the interest operator identifies the locationof the interesting point relative to the barcode.
 4. The method of claim3 wherein the interest operator is selected from the group consistingof: a corner detector; an edge detector; and an area of high texture. 5.The method of claim 1 further comprising the step of: verifying thesimilarity against a threshold value, wherein the threshold value sets amaximum allowed difference in the similarity; and issuing an alert whenthe similarity exceeds the threshold value.
 6. The method of claim 5wherein the alert is selected from the group consisting of: displayingan alert message on a display of the checkout terminal; issuing a visualalarm; issuing an audible alarm; and halting an operation of thecheckout terminal.
 7. The method of claim 1 wherein the step ofcapturing an image of the item comprises: correcting the image into acanonical view, wherein the canonical view is obtained by applying atransform to compensate for an image distortion of the image.
 8. Themethod of claim 7 wherein the transform is identified by comparing asize and shape of the barcode in the image to a predetermined referencesize and shape of the barcode, the predetermined reference size andshape being stored at a storage unit being accessible to the computer.9. The method of claim 8 wherein the size and shape of the barcode inthe image is determined by at least one of: fitting a bounding boxaround the barcode; identifying at least two corner points of thebarcode and relations between the at least two corner points; andcalculating a parameter of a plane on which the barcode resides.
 10. Themethod of claim 7 where in the transform is selected from the groupconsisting of: scale transform, rotational transform, projectivetransform, and barrel transform.
 11. The method of claim 1 wherein thestep of determining a similarity between the visual signature and amodel visual signature comprises: collecting a group of visualsignatures from the item over time; and creating a probabilistic modelof the group of visual signatures to obtain the model visual signatureof the item.
 12. The method of claim 11 wherein the group of visualsignatures of the item is updated by adding the obtained visualsignature to the group of visual signatures.
 13. The method of claim 1wherein the model visual signature is a probabilistic model of a groupof visual signatures obtained from the item over time, the group ofvisual signatures being updated by adding the obtained visual signatureto the group of visual signatures if no fraudulent activity is detectedbased on the obtained visual signature.
 14. A checkout system fordetecting a fraudulent activity, comprising: a computer; a barcodescanner in communication with the computer, the barcode scannerpositioned to detect an item identifier number of an item from abarcode, the barcode being affixed to the item; a camera incommunication with the computer, the camera positioned to capture animage of the item, wherein the image comprises the barcode and asurrounding of the barcode; and the checkout system in communicationwith a storage unit via a network, wherein the checkout system isconfigured to: obtain a visual signature from the image, wherein thevisual signature is obtained based on the barcode and an interestingpoint located in the surrounding of the barcode; determine a similaritybetween the visual signature and a model visual signature by comparingthe visual signature obtained from the image to the model visualsignature associated with the item identifier number, the model visualsignature representing an expected visual signature of the item, whereina plurality of model visual signatures is stored at a storage unit beingaccessible to the computer, each of which being associated with an itemidentifier number; and detect the fraudulent activity based on thedetermined similarity.
 15. The checkout system of claim 14 wherein thebarcode scanner is at least one of an optical-based scanner and alaser-based scanner, the optical-based scanner comprising the camera.16. The checkout system of claim 14 further comprising an alert system,wherein the alert system issues an alert when the similarity exceeds athreshold value, wherein the threshold value sets a maximum alloweddifference in the similarity.
 17. The checkout system of claim 14wherein the checkout system is configured to obtain the visual signaturefrom the image by detecting the interesting point from the image usingan interest operator, wherein the interest operator identifies thelocation of the interesting point relative to the barcode.
 18. Thecheckout system of claim 14 wherein the checkout system is configuredto: collect a group of visual signatures from the item over time; andcreate a probabilistic model of the group of visual signatures to obtainthe model visual signature of the item.
 19. The checkout system of claim18 wherein the group of visual signatures of the item is updated byadding the obtained visual signature to the group of visual signatures.20. The checkout system of claim 14 wherein the model visual signatureis a probabilistic model of a group of visual signatures obtained fromthe item over time, the group of visual signatures being updated byadding the obtained visual signature to the group of visual signaturesif no fraudulent activity is detected based on the obtained visualsignature.